Indirect Reinforcement Learning with Adaptive State Space Partitions

Model-based reinforcement learning can be applied to problems with continuous state spaces by discretizing the spaces with crisp or fuzzy partitions. The manual definition of suitable partitions, however, is often not trivial, since fine partitions lead to a high number of states and thus complex discrete problems, whereas coarse partitions can be unsuitable for the representation of the optimal strategy. In this article a novel model-based reinforcement learning approach, Adaptive Fuzzy Prioritized Sweeping (A-F-PS), is presented. The key idea of the approach is to represent internal models by clustered transitions. From clustered transitions suitable partitions of the state space can be easily derived. Moreover, discretized models corresponding to these partitions can be easily calculated. Clustering is performed with an incremental variant of the fuzzy c-means algorithms, such that transitions need not to be explicitly stored and the A-F-PS approach therefore has moderate storage requirements. The effectiveness of the method is shown by an example from traffic signal control.